Skip to content

[mosaic] Reject TIMESTAMP(0) when validating Mosaic data fields - #11241

Closed
LuciferYang wants to merge 1 commit into
apache:masterfrom
LuciferYang:fix/mosaic-timestamp-precision
Closed

LuciferYang wants to merge 1 commit into
apache:masterfrom
LuciferYang:fix/mosaic-timestamp-precision

Conversation

@LuciferYang

Copy link
Copy Markdown
Contributor

Purpose

close #11228

Precision 0 maps to the Arrow SECOND unit, which the Mosaic native writer does not accept. A Mosaic table with a TIMESTAMP(0) or TIMESTAMP(0) WITH LOCAL TIME ZONE column can be created, but every write fails with writer open failed: unsupported Timestamp unit: Second. MySQL CDC creates such columns from DATETIME.

MosaicFileFormat.validateDataFields now rejects precision 0, also inside ARRAY and MAP, with the same message style as the other types Mosaic rejects, and suggests TIMESTAMP(3).

This check also runs when a table is loaded. An existing Mosaic table with a TIMESTAMP(0) column could not be written before and now cannot be read either, until the column is changed to TIMESTAMP(3) or file.format is set back. Both ALTERs work on such a table.

Tests

MosaicFileFormatTest:

  • testValidateDataFieldsTimestampPrecisionZeroUnsupported checks the full message for TIMESTAMP(0), ARRAY<TIMESTAMP(0)>, TIMESTAMP(0) WITH LOCAL TIME ZONE and MAP<STRING, TIMESTAMP(0) WITH LOCAL TIME ZONE>. It fails without the change.
  • testValidateDataFieldsTimestampPositivePrecisionSupported checks that precisions 1 and 9 of both types are still accepted.

@JingsongLi

Copy link
Copy Markdown
Contributor

This addresses the same writer limitation in the same schema-validation path as the earlier #11235. I verified that the two changes reject the same precision-zero types, including when the visitor recurses into a collection. The 162 Mosaic tests passed on the merge with current master, including the new nested cases. Closing this duplicate so #11228 can be resolved in one PR. Please carry the nested ARRAY/MAP cases into #11235.

One upgrade scenario matters for the retained fix. I wrote a row to a Mosaic table and added a nullable TIMESTAMP(0) column using master. Master still returned the existing row with NULL in the new column. With this patch, that same read fails during table initialization, before opening any data file. The retained fix should reject unsupported new definitions or writes while preserving reads of existing data.

@JingsongLi JingsongLi closed this Nov 19, 2026
@LuciferYang

Copy link
Copy Markdown
Contributor Author

Thank you @JingsongLi

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] Mosaic accepts TIMESTAMP(0) columns that its native writer cannot write

2 participants